1.1.1 Importation du fichier csv

store <- "data/mediacloud"
  media <- "fr_BEN_tribun"
  type <-".csv"
  
  fic <- paste(store,"/",media,type,sep="")
  
  df<-read.csv(fic,
               sep=";",
               header=T,
               encoding = "UTF-8",
               stringsAsFactors = F)
  
  # eliminate duplicate
  df<-df[duplicated(df$title)==F,]
  
  
  kable(head(df))

1.1.2 Resolution of encoding problems

It is sometime possible to adapt manually the encoding problem whan they are not too much as in present example.

1.1.3 Transformation in quanteda format

We propose a storage based on quanteda format by just transforming the data that has been produced by readtext. We keep only the name of the source and the date of publication.

# Create Quanteda corpus
  qd<-corpus(df,docid_field = "stories_id")
  
  
  # Select docvar fields and rename media
  qd$when <-as.Date(qd$publish_date)
  qd$who <-media
  docvars(qd)<-docvars(qd)[,c("who","when")]
  
  
  
  
  # Add global meta
  meta(qd,"meta_source")<-"Media Cloud "
  meta(qd,"meta_time")<-"Download the 2021-09-30"
  meta(qd,"meta_author")<-"Elaborated by Claude Grasland"
  meta(qd,"project")<-"ANR-DFG Project IMAGEUN"

store <- "data/mediacloud"
  type<- ".RDS"
  myfile <- paste(store,"/",media,type,sep="")
  myfile

[1] "data/mediacloud/fr_BEN_tribun.RDS"

saveRDS(qd,myfile)
  qd[1:3]

Corpus consisting of 3 documents and 2 docvars.
  1128961138 :
  "Bénin : les vœux de Patrice Talon aux béninois"

  1129273235 :
  "RDC : l'ultimatum des USA et de l'UE à Joseph Kabila"

  1129306754 :
  "Ali Bongo paralysé? la rumeur lancée par un site panafricain"

summary(qd,3)

Corpus consisting of 20639 documents, showing 3 documents:

         Text Types Tokens Sentences           who       when
   1128961138     9      9         1 fr_BEN_tribun 2019-01-01
   1129273235    11     11         1 fr_BEN_tribun 2019-01-02
   1129306754    11     11         2 fr_BEN_tribun 2019-01-02

1.1.4 Back transformation to tibble

In the following steps, we will make an intensive use of quanteda, but sometimes it can be useful to export the results in a more practical format or to use other packages. For this reasons, it is important to know that the tidytextpackage can easily transform quanteda object in tibbles which are more classical and easy to manage and to export in other formats like data.frame or data.table.

1.2.1 Load dictonary

We start by loading the last version of the Imageun dictionary and we extract our target language (here : french).

   ISO3           x lang
  1:  ABK     abkhaz*   fr
  2:  AFG     afghan*   fr
  3:  AFG      kaboul   fr
  4:  AFG afghanistan   fr
  5:  AGO      angola   fr
  6:  AGO   angolais*   fr

1.2.2 Load corpus

1.2.3 Load tagging function

1.2.4 Annotate all entities

In a first step, we annotate all geographic entities together in order to benefit from the cross-definition of their respective compounds. We will separate them by subcategories in a second step.

[1] "Program executed in  1.73629403114319"

      0     1     2     3     4     5 
  13089  6430   992   112    15     1 

1.2.5 check news with maximum state number

      0     1     2     3     4     5 
  13089  6430   992   112    15     1 

1.3.1 Dictionary

We decide here to use lower case transformation. We use a star for the words that can take a plural form.

1.3.2 Annotation

1.3.3 Visualization

1.3.4 Save thematically anotated corpus

2.1.1 Statistical table

Whatever the dimensions we decide to cross, we build a table where we realize a statistical test in order to identify the cells that are characterized by positive or negative outliers i.e. cells where the phenomena of interest (WHAT) is significantly more present or less present than usual. More precisely, the function will produce two for each cell of the cross dimension table :

• a salience index (Xobs/Xest) : defined as the ratio between observed and estimated number of news where the topic is present.
• an outlier index (prob (Xobs > Xest)) : defined as the probability that the number of news where the topic is present is significantly greater than expected.

In both cases we introduce two parameters of control that will limit the computation of indexes to the cells where it appears statistically relevant to realize the measure :

• Minimum sample size (minsamp) : is the total number of news present in the cell before to compute the probability of apparition of the topic. The default value is equal to 20 as we consider as not meaningfull to compute a proportion on a smaller sample.

• Minimum estimated value (mintest): is the threshold of computation of the chi-square test according to the estimated number of news where the topic is present. According to statistical rules of the chi-square test, this threshold should be equal to 5 for optimal conditions of application. The package R introduce indeed a warning message if the condition is not satisfied, which can increase the time of computation.

Of course, the user can decide to relax or reinforce these two conditions but it is normally better to avoid to do it. When conditions are not fulfilled, the graphic output will not display the cells where the indexes can not be computed.

The function that realize the test is the following one

2.1.2 Interactive graphic

Once the statistical table has been computed, the user can choose between two different visualizations, based on the salience index (exploration) or the chi-square test (ouliers detection). In both case the result will be an interactive figure realized in plotly where it is possible to click on each cell and have a look at the statistical parameters.

The user interested in static graphic (e.g. for publication) can easily adapt the program and realize new functions, for example in ggplot2.

In order to illustrate each type of graphic, we will choose the example of the topic of mobility without distinction between migrants and refugees.

2.2.1 Function

2.2.2 Example

   what  news pct
  1:   NA 20639 100

2.3.1 Function

2.3.2 Example

We present here the statistical table and the two types of graphics that can be produced. In the following case we will only present the outlier graphic.

The analysis reveal a clear over-representation of the topic in the french newspaper Le Figaro (4.37% of news) as compared to the other media (2.1 to 2.5%).

2.4.1 Function

2.4.2 Example 1 : 2014-2015 by month

The analysis reveals clear discontinuities in the timeline of the topic. We start with a low level (0.5 to 1.2%) from January 2014 to March 2015, followed by a brutal jump in April-June 2015 (3 to 5%) and a major peak in september 2015 (15.8% of news). At the end of the period, the level is clearly higher than at the beginning.

2.5.1 Function

2.5.2 Example

When we realize the map, we eliminate the news related to the topic where no countries has been mentioned. As a consequence the reference value is modified : in the whole sample 2.73% of news was related to the topic but in the sample of news where one country is mentioned 2.83% of the news are related the topic.

As the total number of news can be small in some countries, we have reduced here the parameters of the statistical test in order to visualize more countries on the map. It is therefore necessary to be cautious in the analysis of results.

The analysis reveals that some countries are “specialized” in the topic during the period of observation. For example 53.5% of the news about Hungary was associated to the question of migrants and refugees, which is obviously related to the mediatization of the wall established by Viktor Orban in 2015. Other countries are characterized on the contrary by an under-representation of the topic like the USA where the topic is only associated to 0.7% of news. But the situation will change after Donald Trump’s election who will also establish a wall which will dramatically increase the number of news about USA and migrants.